DOCUMENT RESUME 



ED 459 848 

AUTHOR 

TITLE 

SPONS AGENCY 
PUB DATE 
NOTE 



CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



ABSTRACT 

dissemination and reuse of educational resources has sparked a number of 
large-scale digital library efforts. This article describes usability 
findings from one such project- -the Digital Library for Earth System 
Education (DLESE) -- focusing on its role in the process of educational 
resource reuse. Drawing upon a reuse model developed in the domain of 
software engineering, the reuse cycle is broken down into five stages: 
formulation of a reuse intention; location; comprehension; modification; and 
sharing. Using this model to analyze user studies in the DLESE project, 
several implications for library system design and library outreach 
activities are highlighted. One finding is that resource reuse occurs at 
different stages in the educational design process, and each stage imposes 
different and possibly conflicting requirements on digital library design. 
Another finding is that reuse is a distributed process across several 
artifacts, both within and outside of the library itself. In order for reuse 
to be successful, a usability line cannot be drawn at the library boundary, 
but instead must encompass both the library system and the educational 
resources themselves. (Contains 38 references.) (Author) 



IR 058 384 

Sumner, Tamara; Dawe, Melissa 

Looking at Digital Library Usability from a Reuse 
Perspective . 

National Science Foundation, Arlington, VA. 

2001-06-00 

lip.; In: Proceedings of the ACM/IEEE-CS Joint Conference on 
Digital Libraries (1st, Roanoke, Virginia, June 24-28, 

2001). For entire proceedings, see IR 058 348. Figures may 
not reproduce well. Also funded by University Corporation 
for Atmospheric Research. 

ATM-9732665; ATM-9978338 

Association for Computing Machinery, 1515 Broadway, New York 
NY 10036. Tel: 800-342-6626 (Toll Free); Tel: 212-626-0500; 
e-mail: acmhelp@acm.org. For full text: 

http : //wwwl . acm.org/pubs/contents/proceedings/dl/379437/ . 
Reports - Research (143) -- Speeches/Meeting Papers (150) 

MF01/PC01 Plus Postage. 

Computer System Design; ^Educational Resources; *Electronic 
Libraries; Information Dissemination; Information Services; 
Information Sources; Information Systems; ^Shared Resources 
and Services; User Needs (Information) 



The need for information systems to support the 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



IR058384 ED 459 848 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



D. Cotton 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



Looking at Digital Library Usability 
from a Reuse Perspective 



Tamara Sumner and Melissa Dawe 
Center for LifeLong Learning and Design 
Dept, of Computer Science and the Institute of Cognitive Science 
University of Colorado at Boulder 
+1 303 492 2233 

[sumner, meliss]@ colorado.edu 



omra S ^ PA P TMENT 0F EDUCATION 

Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

A^Tnis document has been reproduced as 
r received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



ABSTRACT 

The need for information systems to support the dissemination 
and reuse of educational resources has sparked a number of large- 
scale digital library efforts. This article describes usability 
findings from one such project - the Digital Library for Earth 
System Education (DLESE) - focusing on its role in the process 
of educational resource reuse. Drawing upon a reuse model 
developed in the domain of software engineering, the reuse cycle 
is broken down into five stages: formulation of a reuse intention, 
location, comprehension, modification, and sharing. Using this 
model to analyze user studies in the DLESE project, several 
implications for library system design and library outreach 
activities are highlighted. One finding is that resource reuse 
occurs at different stages in the educational design process, and 
each stage imposes different and possibly conflicting 
requirements on digital library design. Another finding is that 
reuse is a distributed process across several artifacts, both within 
and outside of the library itself. In order for reuse to be successful, 
a usability line cannot be drawn at the library boundary, but 
instead must encompass both the library system and the 
educational resources themselves. 
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1. INTRODUCTION 

Science educators have repeatedly called for information systems 
that can effectively deliver quality educational materials in 
formats that are readily accessible, with a high degree of 
confidence that the materials will be useful, interesting, and 
effective [27]. This has largely been interpreted as a call for 
digital library systems or information systems with library-like 
services [16]. Towards this end, there are many digital library 
efforts underway aimed at improving the quality of undergraduate 
or K-12 science education [7, 23, 24]. Typically, these systems are 
similar to web portals, providing access to a managed collection 
of community-constructed educational resources with services for 
resource discovery, and possibly peer-review and resource 
creation as well. One prominent example of such a system is the 
NSDL (National Science, Mathematics, Engineering, and 
Technology Education Digital Library), a program initiated by the 
National Science Foundation to integrate multiple digital libraries 
in these areas and provide access to a broad variety of educational 
materials [36]. These efforts are based on the assumptions that 
providing digital libraries of educational resources can [21]: 

• Improve the quality of education by promoting the reuse of 
educational resources that are proven to be effective; 

• Improve the productivity of faculty through resource reuse 
and sharing; 

• Help foster an active community of learning and innovation, 
where best practices and resources are developed and shared. 

The underlying belief is that the reuse of existing resources or 
knowledge to create new educational resources will lead to 
improvements in both product (better educational resources) and 
process (teaching and learning). These assumptions and beliefs 
mirror similar goals for shared resource collections, henceforth 
referred to as ‘repositories,’ in both the business world and the 
software engineering community. In the business world, much 
recent activity has focused on using shared repositories to support 
‘knowledge management’ and ‘best practice’ sharing to improve 
organizational efficiency and foster innovation [1, 17, 35]. The 
software engineering community has been vigorously promoting 
code sharing and software reuse for years since reuse is linked to 
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measurable improvements in both programmer productivity and 
software quality [8, 10,30,31]. 

In the field of software engineering, reuse has been defined as the 
utilization of pre-existing components to ultimately create 
something new [10]. We feel that this is an appropriate definition 
for reuse of educational resources as well. For software 
developers, the product created might be an application; for 
educators it is a class or a curriculum. Although the products 
differ, we posit that the process of reuse in these two domains is 
essentially the same. Thus we feel that important lessons can be 
drawn from examining software engineering models of reuse: in 
particular, what promotes and what inhibits successful reuse. 

Currently there is little research on the effects of reuse on 
learning. In literature from the library community [25], it has been 
observed that digital libraries can affect learning in different 
ways: through direct student interaction with the digital library 
system, and through the utilization of digital library resources by 
educators in their classroom. Studies show a positive correlation 
between learning and student interaction with a digital library 
[25]. Less is known about the relationship between educator reuse 
and learning. However, it is important to note that reuse is not an 
activity new to educators. On the contrary, the reuse process is 
deeply embedded in the way educators work today, as they locate, 
utilize and share physical as well as digital resources. In theory, 
digital libraries should support these current practices and extend 
them further, by making available a richer variety of educational 
resources, at all levels of granularity: ranging from single images, 
lesson plans and applets, to lab modules, to entire courses. Ideally, 
such facilities could enable educators to have more creativity and 
control of the materials they use in the classroom. 

Past experiences in software engineering suggest that while a 
library is essential for supporting systematic reuse at the 
institutional level, simply providing a library of resources is 
insufficient to guarantee that effective reuse will occur at the 
individual level: 

although the library metaphor has guided early work in 
classification, storage systems, and other areas of reuse 
technology, it does not provide the best focus for setting up 
and running a reuse program. [...], it has simply not yielded 
a major change in the way most people develop software. 
Poulin, page 1 , [30]. 

One survey on the state-of-the-practice in reuse found that (1) 
taking the library approach to software reuse and (2) the 
effectiveness or efficiency of the software library were two of the 
most significant raw-predictors of reuse capability in an 
organization [31]. These results are quite disturbing given the 
current emphasis on ‘digital libraries’ as the primary means for 
facilitating the reuse of educational resources, and ultimately for 
changing the way that educators develop teaching resources. 

In this article, we examine existing studies and theories on 
resource reuse from the software engineering literature. We 
believe that this prior research has much to offer digital library 
projects that are deeply concerned with fostering reuse within 
their communities of users. In particular, we develop and discuss a 
cognitive model of resource reuse derived from the software 
engineering literature - the location, comprehension, 



modification, and sharing cycle - and compare this model with 
existing digital library information lifecycle models. We focus 
mainly on comprehension and modification processes, as they are 
critical yet relatively understudied aspects of digital library use. 
We use this model to critically examine our own experiences with 
the Digital Library for Earth System Education (DLESE) project. 
This reformulation of the problem from ‘providing a library’ to 
‘supporting reuse’ has significant implications for system design, 
community outreach and training, and future research directions, 
which we will discuss. We will first briefly present this model of 
reuse before proceeding to the analysis. 

2. A MODEL OF REUSE 

Reuse is cognitive activity that is embedded in an overall task- 
directed design process; as described above, the person’s primary 
goal is to create something new. For instance, a faculty member 
may want to design a new lecture or even a new course, and may 
consider combining and reusing existing resources (e.g., lesson 
plans, exercises, maps, data sets) as part of this overall process. 
Reuse involves both composing new resources from existing 
resources and developing resources that can be reused in the 
future [9]. Previous studies of software programmers [8] suggest 
that when composing new resources from existing resources, 
reuse involves three closely intertwined cognitive activities: 
location, comprehension and modification (Figure 1). 

As shown in Figure 1 , the first step in the reuse process is forming 
a ‘reuse intention’; i.e., deciding to use the resource repository in 
the first place. Ye et. al. outline numerous ways the reuse process 
can fail at this first step [38]. Software developers can be reluctant 
to disrupt their workflow and endure the cognitive effort it takes 
to switch back and forth between their development environment 
and the repository. Developers also chronically. underestimate the 
amount of time it takes to develop from scratch, which leads to a 
cognitive bias against deciding to reuse in the first place. Ye 
argues that these behaviors stem from loss aversion tendencies in 
human decision-making processes; i.e., people are more sensitive 
to avoiding potential losses (like wasting time looking for 
resources that don’t exist) than to realizing potential gains 
(finding a useful resource). 

Location refers to the process of finding potentially useful 
resources in the resource repository. The user must be able to 
translate his or her reuse intentions (their situation model) into 
appropriate system queries (the system model) [12]. Numerous 
studies indicate the many people, not just software developers, 
have difficulties with this; Norman referred to this challenge as 
the gulf of execution and evaluation [26]. 

In reuse, location is tightly intertwined with comprehension. 
Comprehension involves not only making relevance judgments, 
but also understanding the function, structure, and context of use 
of the resource in order to decide if it needs modification and how 
to go about making the necessary modifications. Unsurprisingly, 
research has found that people are often unwilling to take the time 
to thoroughly comprehend a resource, and instead employ 
“comprehension avoidance” strategies [9]. For example, a 
developer might execute a piece of software to see what it does, 
rather than read all its documentation. Often times, developers just 
abandon the reuse process at this point. 
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Typically, resources cannot be used ‘as is’ but instead must be 
modified before they can be used. Modification can take many 
forms. Studies of the evolution of resources in software 
repositories show that software components evolve through four 
main processes: refinement, composition, abstraction, and 

factorization [9]. These processes will be defined and discussed in 
the following section. However, it is sufficient to note that many 
people drop out of the reuse cycle here because they are either 
unable to make the necessary changes (because they lack the 
technical skills or an adequate understanding of the resource), or 
are unable or unwilling to spend the necessary time. 

In this context, ‘sharing’ refers to the resource producer making a 
conscious decision to make a new resource available to others to 
reuse and undergoing the work necessary to do so. Sharing 
resources is a crucial part of an effective and sustainable reuse 
cycle, but it is often viewed as being an extra step that is outside 
of the overall design process. Bannon and Bodker discuss the 
extra work it takes to place items into a shared resource repository 
[3]. They argue that both the resource producer and the resource 




Figure 1: Location-Comprehension-Modification- 
Sharing Cycle (L-C-M-S) 



consumer must make a conscious effort to understand each other’s 
context of use and that reusability is enhanced when the producer 
can anticipate the consumer’s context of use to some degree and 
makes the effort to ‘package’ aspects of this context with the 
resource. In software engineering, studies show that this 
packaging for reuse takes a significant effort and can double the 
production costs of more complex resources [10]. Many 
organizations have found it necessary to implement reward 
programs, often involving direct monetary compensation, to 
encourage developers to do the extra work required to submit 
resources to the repository [30]. 

In the following section, we will use this model to examine some 
of our experiences in the DLESE project, particularly noting the 
similarities and differences in the observed reuse behaviors of 
undergraduate teaching faculty and software developers. 



3. CASE: DIGITAL LIBRARY FOR EARTH 
SYSTEM EDUCATION 

DLESE is a grassroots, community-led project to provide 
searchable access to high-quality, online educational resources for 
K-12 and undergraduate earth system science education. These 
educational resources include objects such as maps, simulations, 
lesson plans, lab exercises, data sets, virtual field trips, etc. These 
resources are created by either individual faculty members or by 
institutions (e.g., NASA, USGS) and are held (stored) on their 
local servers. When resources are contributed to the library, either 
the resource creators or DLESE catalogers create metadata 
describing the resource to support resource discovery. The 
metadata scheme, based on the IMS LOM standard [14], is quite 
detailed and is fully described elsewhere [11]. This metadata is 
centrally stored and managed and is, in effect, the chief holding of 
the library. 

As detailed in the DLESE Community Plan [19], the success of 
DLESE will ultimately be measured in two ways. Firstly, it will 
be measured by the library’s impact on earth system science 
teaching and learning. Is student learning improved? Do faculty 
teach more effectively or efficiently? Both of these outcomes 
depend on faculty effectively reusing the resources available in 
DLESE in their own teaching. Secondly, success will be measured 
by the sustainability and viability of the community contribution 
process. Do faculty create reusable resources to share with others? 
Can they create the necessary metadata to help their colleagues 
discover and use their resources? Thus, success in DLESE heavily 
depends on faculty participating deeply in a culture of reuse in 
terms of both resource design and teaching practices. 

For the past 18 months, the project has focused on building and 
evaluating a library prototype, designing a resource review 
process, and setting up the community governance structure [23]. 
Part of the current library interface is shown in Figure 2. Faculty 
can search by keyword and grade level to locate resources. The 
search results are presented as a list of information summarizing 
the potentially relevant resources such as resource title, brief 
description, grade level, etc. Clicking on either the resource title 
or URL will bring up the educational resource directly in a new 
browser window. 

During this period, a number of formative studies were conducted 
to help us understand community needs and to evaluate the 
evolving library prototype. As part of the requirements analysis 
process, workplace interviews were conducted with seven earth 
science faculty and two students. Participants were selected that 
were already using digital resources in their teaching and learning. 
Each interview lasted between 60 to 90 minutes and was taped, 
transcribed, and analyzed. The focus of the interviews was to 
understand in detail how participants located and selected digital 
educational resources. We found that there is an important 
distinction in the way instructors prepare for a course, and for a 
class, which suggest that these two processes must be treated 
differently (Table 1). 
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Figure 2. Discovery system interface in DLESE. The search interface is in the upper left 
corner. The search results page is shown in the middle. Clicking on ‘View Full Record’ 
brings up the full metadata record, which includes a short description of the resource. 



When preparing for a class, educators are searching for items that 
will immediately plug in to the existing framework of their 
curriculum. In this task, reuse occurs downstream in the design 
process, where most of the overall design has been done, and so 



Table 1. Two example DLESE tasks. 



Prepare for a Class 


Prepare for a Course 


Jeff has an hour and a half 
before his class on 
environmental issues of 
greenhouse gases and ozone 
depletion. He wants to spice up 
his lecture. He browses online 
repositories he knows for 
pictures, charts, animations, or 
interactive tools to use during 
the class. He wants to conduct a 
pre-screening of sites for 
suitability and note the 
locations of these materials for 
his students, so they can study it 
further. The materials he uses 
must be from a source he trusts 
and be useable without a lot of 
alteration, since there is little 
time. 


Kim is an introductory earth 
science teacher. She wants to 
teach her students about deep 
time, and how climate has 
varied over time in one 
location. She knows this is a 
difficult concept. She is 
looking for resources that can 
educate her in the area, as well 
as some pedagogical tools. In 
particular, she needs some 
help teaching students how to 
understand and get past their 
problems with "deep time." 
She would like to locate 
tutorials in this area, and 
experts in the field who she 
might be able to contact for 
teaching ideas. 



the search is highly targeted and specific according to the existing 
design. At this point educators are usually working under a severe 
time constraint, and don’t have time to digest large amounts of 
information or adapt items for use. Thus they are looking for 
resources that are as context independent as possible, so that they 
require little adaptation effort. Such items include images, 
diagrams, vocabulary lists, and other resources that have minimal 
environmental or other dependencies. 

This type of preparation is different from the work an educator 
goes through to prepare for a course. In this task, reuse plays a 
role upstream in the design process, in that much of the structure 
of the course itself is in a formative stage. Preparing for a course 
is generally on the time scale of weeks or months, rather than 
minutes or hours. In this process instructors appear to be more 
willing to explore new resources and test them for their 
classroom. They may have a vague idea of what they are looking 
for, and are interested in browsing collections to gain ideas. They 
can afford to take more time to synthesize larger resources and 
adapt them to their classroom, or design curriculum around them. 

One of the challenges of designing a digital library is to support 
these two processes simultaneously, which can create tensions and 
conflicts in the system requirements. To analyze how well the 
evolving DLESE prototype supports these two key user tasks, we 
conducted two rounds of usability testing with earth science 
faculty (one with five participants and one with ten). Each 
participant was asked to think aloud as he or she performed a 
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series of class and course preparation tasks using the current 
DLESE prototype. Detailed observational notes were 
independently taken by two observers and compared after each 
session. Additionally, we analyzed the structure and content of 
some of the resources with an eye towards developing heuristics 
for creating metadata that best facilitates location and resource 
comprehension. We now present the results of these studies and 
analysis in terms of the L-C-M-S reuse cycle. Table 2 summarizes 
the results for each of the two types of tasks. 

3.1 Formulation of reuse intentions 

Our research has shown that faculty formulate reuse intentions, as 
exemplified in the above reuse tasks taken directly from user 
interviews. We discovered that their primary source for 
identifying materials for reuse is through personal interaction of 
some kind: they ask colleagues for suggestions, they exchange 
ideas and resources at workshops and conferences, they even go 
to the library and ask the librarian for known sources. A crucial 
aspect of this process is the reliance on a trusted source for 
resource location and evaluation. Faculty appear to use the source 
(either the individual creator, or an institution) of the material as 
the primary way to determine its quality and effectiveness. 

The Internet plays a role for many faculty in the reuse process, 
though this is highly dependent on the conditions of access and 
other environmental factors. However even in the best cases of 
Internet availability, we found that faculty do not commonly rely 
on search engines to locate resources, but rather go to sites that 
they already know and trust. Examples of trusted sites include 
colleague’s personal home pages, or known organizations’ 
websites, like NASA, USGS, or Discovery.com. In one case, a 
faculty member actually walked to the library and asked the 
librarian to search the web for a particular item, and then scribbled 
down the URL to later type in to his browser! 

Faculty resist “cold” searching the web for a number of reasons: 
the thousands of hits returned by search engine queries and the 
time required to evaluate them; the frequently unrelated or low 
quality websites returned from a search query; and the many links 
that are broken or no longer point to relevant material. In short, 
web searching is perceived as inefficient and frustrating because 
of the enormous amount of information to sort through and high 
variability of quality of results. 



The observation that instructors are only willing to visit and 
search trusted sites suggests that quality filtering for resources 
occurs mainly at the location stage, not at the comprehension 
stage as one might guess, in that faculty are resistant to even 
search sites with which they are not familiar. This finding echoes 
results from software engineering which found that while the 
perceived quality of the repository as a whole is an important 
factor, the certification of individual resources to some quality 
level appears to have no measurable impact on reuse [31]. 

Research has shown that an important factor in successful web 
searching is knowledge not only in the domain of the topic being 
searched, but also in techniques of web searching itself [13, 20]. 
In general, we observed that most faculty have a fairly low level 
of expertise in web searching, most likely due to lack of training. 
There is evidence that effective searching and evaluation are skills 
that need to be taught, regardless of the searcher’s domain 
knowledge [15]. The most common search technique we observed 
for faculty was to type one or two keywords into a text box 
(although one faculty member claimed to always compose queries 
of one word or less). Many faculty did not recognize a difference 
between different search engines, and expressed frustration when 
different search engines returned divergent results to the same 
query. The frustration felt is equally due to the limitations and 
often minimal query support offered by current web search 
engines, which in many cases only provide a single search box 
with no instructions for query formulation. 

When considering the World Wide Web as a repository, we have 
found that there is a gap between the formulation of reuse 
intentions, and the location of effective material. We have 
evidence from the interviews that faculty are forming reuse 
intentions, but these intentions are not being translated into an 
effective search strategy. The implication of this finding for the 
design of DLESE is two-fold: the system must provide better 
search capabilities than traditional search engines, and it must 
encourage and educate faculty to utilize this functionality. In other 
words, part of the role of DLESE is to help educators translate 
their intentions into an effective search query that will lead to the 
location of desired resources. 

3.2 Location 

The first step to successful location of resources through DLESE 
is to ensure that faculty know and accept DLESE as a trusted site. 
To make DLESE known in the community of earth science 



Table 2. Examining two DLESE tasks from a reuse perspective. 





Forming reuse 
intentions 


Location 


Comprehension 


Modification 


Sharing 


Preparing for 
a class 


Downstream: 
Specific, well- 
defined 


Highly 

targeted 


Minimal time and effort available 
Comprehension avoidance 


Little or none: Bookmark 
lists; Factorization of 
complex resources 


Relatively easy 


Preparing for 
a course 


Upstream: 
Broad, loosely 
defined 


Maybe 

explorative 


More time and effort available 
Metadata records important 

Supplementary aids in resources 
(tables of contents, indexes, etc.) 
necessary 


Modification likely: 
Composition of resources; 
Refinements 


More effort 
required for 
‘packaging’ 
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educators, the project has actively engaged the community in 
many aspects of DLESE development. It has also undergone 
extensive outreach efforts, which are described more fully in the 
DLESE Community Plan [19]. 

In order to best support our diverse audience, the DLESE resource 
discovery system has been designed to be both simple and 
powerful. One of the important advantages of DLESE over 
traditional search engines is that the resources are well described 
with a detailed metadata scheme, and therefore can be searched 
for in more powerful ways. The discovery system supports both 
searching and categorical browsing. These two search methods 
have been fully described and compared elsewhere [4, 37], and it 
has been shown that users often use a combination of both 
processes in the discovery process and it is important to support 
both. Browsing, in particular, supports search where the query is 
vague or explorative in nature, which may be the case in the task 
of preparing for a course. 

In addition to entering keywords, the discovery system also allows 
the user to limit the search by specifying various search 
parameters (type of resource, grade level, computer requirements, 
etc.). In order to maintain a simple and easily usable discovery 
system, DLESE provides two different interfaces: a simple and 
advanced search. The simple search interface, shown in Figure 2, 
is similar to a common search engine and thus familiar to many 
users. It provides a text box for keywords and allows the user to 
specify the grade level of the resource. The advanced search 
interface supports other search parameters, such as educational 
resource type, and can be used by more advanced users. 

After a search is executed, a results page is shown with a listing of 
the resources matching the search query. Along with the URL, 
some key information is shown for each resource: the title, brief 
description, grade level, and resource type. The quality and 
quantity of information presented at this stage is crucial both for 
resource location and comprehension, as described below. The 
user is also provided a link to the full metadata record for each 
resource. In preliminary studies, we have found that users 
frequently utilize the metadata record when evaluating the 
relevancy before visiting the resource itself, demonstrating that 
the two processes of location and comprehension are tightly 
intertwined. 

3.3 Comprehension 

As mentioned above, the comprehension of a resource begins with 
the search results page, so it is important that the results page 
provide the right type and right amount of information. In 
preliminary testing, we observed comprehension avoidance 
strategies at this stage, in the form of keyword scanning or 
skipping over text entirely, when the amount of information 
presented was considered too great. Interestingly, we found that 
there was not a graceful degradation of comprehension at this 
point, but rather complete failure because the text was ignored 
entirely if it was too long! 

The design of the results page is one case where there exists a 
tension between supporting the two tasks described earlier: 
preparing for a course and preparing for a class. In the former, the 
instructor may want more detailed resource descriptions and 



secondary information, like suggested variations of use, related 
material, etc. In the latter case, where support for extremely rapid 
evaluation of a resource is essential, keywords and brief 
descriptions are ideal. These conflicting goals must be considered 
not only in the design of the search results page, but everywhere 
that comprehension takes place. This includes the presentation of 
the metadata record, and even the resource itself, as described 
below. 

Possibly the most important finding is that comprehension is a 
distributed process across several artifacts, both within and 
outside of DLESE. Indeed, we observed instructors go back and 
forth between the search results page, the metadata description 
page for a particular resource, and the resource itself in the 
process of comprehension. This implies that in order to support 
reuse, all of these components must be considered. 

A frequent and unexpected cause of failure at this stage was a 
granularity mismatch between what was promised by the metadata 
and what was actually displayed at the resource URL. For 
example, in one case an instructor restricted his search query to 
search only for lesson plans. The results page returned links to 
large educational sites that had numerous holdings, or entire 
courses that had lesson plans embedded in them. This was a 
frustrating and ultimately unsuccessful search experience for the 
instructor, as he expected to be taken to a lesson plan and instead 
was taken to a site where he had to search further. In many cases 
it was not at all obvious that the site was even relevant to the 
query the instructor had constructed. 

An implication of the granularity mismatch problem is that the 
system needs to be designed to better support different levels of 
granularity of large websites. This problem manifests in both the 
comprehension and modification stages. The problem of defining 
what comprises an item vs. a collection in a digital library is an 
entire research topic on its own, but it is worth stating here that 
accurate correspondence between the metadata and the resource it 
points to is essential. To achieve this it may be necessary for 
DLESE to separately classify components of a resource, e.g. key 
images or lessons plans, with their own metadata and thus treat 
them as separate resources. 

As mentioned above, we observed that failure in comprehension 
can occur at the resource itself, regardless of the level of detail or 
accuracy of the metadata. This implies that to be serious about 
reuse, we cannot draw a usability line at the library boundary. 
Usability of the resources held in the library is equally if not more 
important in the success of the reuse cycle as the rest of the library 
system. Thus, an essential function of the digital library is to 
encourage resource creators to create resources that are more 
usable. This can be viewed as a ‘packaging’ problem, which will 
be described in detail in the sharing section. 

3.4 Modification 

At first blush, modifying educational resources might appear to be 
outside of the purview of a digital library system like DLESE. Our 
studies, and experiences of other educational component libraries 
such as the Educational Object Economy project, suggest that 
many educational resources, even simple textual resources, need 
to be modified in some way before they can be used [34]. As 
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mentioned earlier, studies of software repositories have observed 
resources to evolve through four main processes: composition, 
factorization, refinement, and abstraction. We have observed the 
first three processes in our studies, which we will discuss in turn. 

Composition refers to creating a new resource by combining 
existing resources. We observed composition activities in both the 
preparing for a class and preparing for a course tasks. When 
preparing for a class, the instructor is typically looking for a 
resource with fairly specific requirements to plug into their 
existing lecture framework and is unwilling to spend much (if 
any) time in modification activities. However, instructors often 
also desire to create a compilation of pointers to interesting 
‘further reading’ resources for the students to use after class. This 
type of bookmarking or list-creating activity is also observed in 
the planning stages when preparing for course. Interestingly, 
composition activities have received the most attention in the 
research community. There are many digital libraries, such as the 
ACM’s, that provide simple bookmarking and bookmark sharing 
facilities. There are also more ambitious projects aimed at creating 
digital library tools and services to support more advanced forms 
of composition such as Walden Paths (a tool enabling teachers to 
construct linear paths through web resources) [33], Iscapes (a tool 
enabling faculty to create a collection of mixed media resources to 
share with their students) [28], and ESCOT (a set of tools and 
services which help faculty and software developers to combine 
interactive educational components without programming) [32]. 

Factorization refers to creating new resources by partitioning 
more complex resources into simpler parts that can be more easily 
shared. We observed the need for factoring in the granularity 
mismatches observed during ‘preparing for a class’ type tasks; 
i.e., the faculty member wants to find a lesson plan or a certain 
image but it is buried in a complex ‘whole course’ web site. There 
are several different approaches that could be taken to support 
factoring. One approach already mentioned would be to support it 
at cataloging time; i.e., create separate metadata records for each 
object in a complex resource. For example, a cataloger might 
break up a course by separately indexing particular units or even 
key images from the course. This is extremely effortful and time- 
intensive, and introduces the challenge of not returning many 
overlapping resources from the same complex object in response 
to a single query. Another approach would be to work with 
resource creators to encourage them to create these sorts of 
supplementary aids as part of the library’s outreach activities. 
Alternatively, tools could be created that factor complex resources 
at comprehension time. For instance, when the user chooses to 
view an online course or other complex resource from the DLESE 
search results page, a computational tool could analyze the 
structure of the course to construct an active table of contents to 
all the main sections and major media elements. This could help 
both comprehension and modification processes by making the 
structure more readily apparent and by quickly identifying simpler 
parts that could be used separately. Our experiences suggest that 
support for factoring could be an important digital library service, 
but to the best of our knowledge there are no library projects 
currently looking at this area. 

Refinement refers to creating a new resource by modifying or 
adding to an existing resource without significantly modifying its 
structure. For simple textual resources, refinements require a 



willingness and ability to engage in basic HTML authoring. For 
interactive resources such as simulations or visualizations, 
refinements could involve programming. Our resource analysis 
suggests that many resources, even simple textual ones, will 
require refinements before they can be used. As a mundane, yet 
pervasive example, we have observed that many educational 
resources cannot be reused ‘as is’ simply because they are overly 
specific; i.e., the contact details and classroom times and locations 
from the last time the resource producer taught are embedded 
directly in the resource. Research into end-user programming and 
authoring tools, such as that begin done in the ESCOT project, are 
looking at how to help faculty refine interactive resources [32]. 
Existing HTML authoring tools such as FrontPage™, 
Dreamweaver™, or even some word processors such as MS 
Word™, could be used to refine textual resources. However, 
many faculty may lack access to these tools, may not know how 
to use these tools, or may simply be unwilling to spend the time to 
switch to another application to make even simple changes. It 
would be fruitful to calculate more precisely the nature of many 
textual refinements; it may be the case that libraries such as 
DLESE could provide a tool with extremely simple editing 
functions that could deal with many of the necessary refinements. 

Abstraction refers to identifying common features across several 
existing resources and creating a new resource that captures these 
commonalities. We did not observe abstraction processes in any 
of our studies or analyses. Our feeling is that abstraction processes 
may occur in more mature reuse communities, where there are 
multiple similar resources to generalize across and there are users 
willing and able to do this type of analysis. 

3.5 Sharing 

As discussed under comprehension and modification, many 
educational resources available on the web haven’t been designed 
with reuse by other faculty in mind, and they are mainly focused 
on supporting students. Several educational publishers have also 
observed this, even for some commercial products: 

“If the needs of the adopting instructor or the adopting 
community are considered along with the needs of the 
student, it will be much easier to transition the innovative 
work that is being done into the classroom at large. " [5] 

Essentially, many faculty are placing resources on the web so that 
they can be used by students (notably theirs), but they are not 
taking the extra step to do the ‘packaging’; i.e., to prepare the 
useful supplementary materials such as tables of contents, 
indexes, summaries, instructors guides, etc that make complex 
resources, such as courses, reusable by other faculty. Even for 
simpler resources like interactive applets, it is rare to find syllabi 
or lesson plans demonstrating how an applet might be used in a 
course [22]. Traditionally, these ancillary materials are developed 
and integrated into the resource as part of the publishing process 
[5, 22]. One challenge for educational libraries like DLESE will 
be to come up with alternative processes that encourage the 
development of such ancillaries. 

At the moment, the metadata created by DLESE staff members 
for library resources is serving as the reuse ‘packaging’ to some 
degree. This metadata includes basic information on the content of 
the resource, technical requirements, and intended users. Thus, the 
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reuse packing is in fact distributed across the library interface and 
the resource itself. In the usability testing, some participants were 
observed to make extensive use of the descriptive metadata 
records as comprehension aids, but then stumble when moving to 
the resource itself because the resource was not as clearly 
structured as the metadata. Other participants did not make use of 
the descriptive metadata, and some became frustrated trying to 
comprehend complex resources. One possible remedy would be 
for DLESE to provide services to resource creators to feed back 
some of the descriptive metadata into the resource itself. For 
instance, once the technical requirements are summarized for the 
metadata it would be fairly straightforward to embed this 
information in a summary section in the resource. 



4. RELATED WORK 

The L-C-M-S model shares some important similarities and 
differences to more general information lifecycle models 
proposed by others [6, 29]. Figure 3 shows the information 
lifecycle model developed by Borgman et. al [6], One important 
difference between the models is that the Borgman model clumps 
location and comprehension into a single activity: searching. We 
believe that it is necessary to distinguish between these two steps, 
because it is important that both are supported. Being able to 
locate good material does not necessarily mean it will be 
comprehended, and the gap between these two stages is frequently 
where reuse fails. In effect, the L-C-M-S cycle is a much more 
specific model that depicts the specific demands of resource reuse 
from an individual’s perspective, whereas the Borgman model 
depicts a general digital library process from the resource’s 
perspective. Interestingly, Paepcke presents a rough model, based 
on the field studies of information needs in an engineering firm, 
consisting of five processes: discover, retrieve, interpret, manage, 
and share [29]. While this model highlights similar processes as 
being important, it does not develop the implications of these 
processes for digital library design and outreach, nor does it 
situate these processes into particular types of user tasks 
specifically, or a reuse process more generally. 

As mentioned in the Introduction, in both software engineering 
and knowledge management, other approaches for supporting 
reuse have proven to be more effective than large, centrally 
managed libraries. Two promising alternatives are the product line 
approach [31] and the small, locally-managed library approach 
[2]. In the product line approach, the organization makes a 
concerted effort to analyze existing products and determine what 
underlying components are shared across products. ‘Packaging’ 
efforts are focused on making these identified components 
reusable, and subsequently, the product line is more easily 
maintained and expanded using these shared components. Perhaps 
an analogous approach could be applied to digital library 
outreach: facilitators could work with an educational program or 
department to identify shared components or opportunities for 
reuse across a suite of courses. In the small library approach, 
repositories are created with a narrow focus on either a specific 
domain (e.g., avionics software) or a suite of related tasks (e.g., 
financial analysis). Typically, these small libraries are locally run 
and managed, which often means that the resource producers and 
the resource consumers are part of the same workgroup. Reuse is 
easier in these cases because producers and consumers share a 
similar use context, which reduces the need for and reliance upon 
‘packaging’. 



CREATIOH | 




Figure 3. Information Life Cycle in Digital Libraries. 
Reprinted from Borgman, et.al. [6] 



5. CONCLUSIONS 

The preceding analysis falls into the broad category of task- 
centered design and analysis, which is a well-known approach to 
understanding software usability [18]. In our case, we examined 
two main tasks - preparing for a class and preparing for a course - 
from the perspective of a reuse model derived from the software 
engineering literature. We have tried to discuss how the 
experiences of faculty using a digital library in their teaching 
share some important similarities and differences to software 
engineers using a shared code repository. One of the key points 
highlighted by this analysis is that for these types of reuse tasks, 
we cannot draw a sharp boundary around the digital library 
system and only concern ourselves with the design and evaluation 
of the library itself. We must take into account the usability of the 
resources as well since the information necessary to complete the 
task, from the user’s perspective, is distributed across the library 
and the resource. Analyzing our experiences from a reuse 
perspective highlighted several possible implications for library 
system design and library outreach activities. 

In terms of system design, the design of the search results page is 
critical for supporting resource comprehension. There must be 
enough information to effectively summarize the resource, but not 
so much text that people skip reading it. Our evaluations show this 
to be a very fine line! Also, the library’s metadata plays a central 
role in documenting the resource enough to support 
comprehension and modification processes. In effect, the 
descriptive metadata is ‘filling in’ for the lack of summarization 
and overview materials in the resources themselves. This 
observation has implications for the use of automatic cataloging 
tools or full-text retrieval methods. For these techniques to replace 
human catalogers creating metadata, they must be considered in 
terms of how well their output supports resource comprehension, 
in addition to supporting resource location. 

The analysis of modification activities suggests that libraries 
could consider providing tools that support factorization and some 
simple refinement activities. While the DLESE community has 
always been interested in providing content creation tools as part 
of the library’s services, modification tools have different needs in 
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terms of functionality and in terms of when such a tool might be 
used. Our analysis suggests that these tools should be available in 
the context of the L-C-M-S cycle; thus they should be easily 
accessible from the search results page and designed to reduce 
context switching between the discovery system, the resource, and 
the tool itself. 

In terms of library outreach, clearly one of the top priorities of the 
outreach effort should be helping potential library users associate 
resource reuse intentions with using DLESE. This is consistent 
with the focus of DLESE’ s current outreach activities. Perhaps 
more surprisingly is the strong need suggested by our analysis for 
outreach activities to work equally closely with resource creators. 
We must take steps to raise the level of the playing field overall in- 
terms of designing and structuring educational resources for reuse 
by other faculty, not just students, and this may be part of the 
library’s overall remit to ‘manage’ the collection. Currently, 
DLESE offers workshops on cataloging and metadata creation to 
interested resource creators. One possibility would be to extend 
these workshops to include a component on designing resources 
for effective reuse, which emphasizes the importance of clear 
labeling, summarizing, and indexing. Additionally, the ‘product 
line’ approach to promoting reuse suggests that outreach efforts 
might consider working with programs or departments, in addition 
to individual faculty members. 

Finally, we tried to demonstrate that certain types of libraries, 
such as DLESE, serve as ‘reuse repositories’ and can benefit from 
prior reuse research in other disciplines. In the software 
engineering discipline, empirical studies have established that 
reuse improves software quality and programmer productivity 

[10]. This discipline has tried a number of approaches for 
fostering reuse in programming organizations, of which libraries 
are just one approach. To the best of our knowledge, no research 
has established the link between educational resource reuse and 
improved student learning or improved faculty productivity. Yet, 
this is the assumption underlying most educational digital library 
projects. Clearly research is needed that looks at whether resource 
reuse does improve teaching and learning, and under what 
conditions. 
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